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ABSTRACT 


Many  neurai  network  models  capable  of  content-addressable  memory  are  shown  to 
be  special  cases  of  the  general  model  and  global  Liapunov  function  described  by  Co¬ 
hen  and  Grossberg  (1983).  These  include  examples  of  the  additive,  brain-state-in-a-box, 
McCulloch-Pitts,  Boltzmann  machine,  shunting,  masking  field,  bidirectional  associative 
memory.  Volterra-Lotka,  Gilpin- Ayala,  and  Eigen-Schuster  models.  The  Cohen-Grossberg 
model  thus  defines  a  general  principle  for  the  design  of  content-addressable  networks.  A 
model- independent  property,  such  as  content-addressable  memory,  that  is  shared  by  all 
model  exemplars  of  such  a  general  design  constitutes  a  computational  invariant.  Such 
a  general  model  and  analytic  method  defines  a  computational  framework  within  which 
specialized  model  exemplars  may  be  compared  to  discover  which  models  are  best  able 
to  explain  particular  parametric  data  about  brain  and  behavior,  or  to  solve  particular 
technological  problems. 


1.  The  Role  of  Design  Principles  and  Computational  Invariants  in  Compu¬ 
tational  Neuroscience 

Due  to  the  highly  interactive  nature  of  brain  dynamics,  the  analysis  of  general  design 
principles,  cellular  mechanisms,  network  modules,  and  functionally  specialized  multi-level 
architectures  have  proceeded  hand-in-hand,  with  each  level  of  analysis  clarifying  the  scien¬ 
tific  understanding  of  the  other  levels.  In  all  of  these  modelling  endeavors,  mathematical 
analysis  and  parametric  computer  simulations  have  played  an  important  role  by  demon¬ 
strating  how  complex  emergent  properties  may  arise  from  interactions  among  simpler  net¬ 
work  components.  Mathematical  analysis  has  also  been  particularly  useful  towards  the 
identification  of  general  design  principles  by  clarifying  how  ostensibly  different  specialized 
model  mechanisms  may  all  be  variations  of  a  general  computational  theme,  in  that  they 
all  exhibit  one  or  more  important  computational  invariants. 

The  mathematical  characterization  of  such  a  computational  invariant  greatly  facilitates 
a  finer,  comparative  analysis  which  cuts  across  the  specialized  models  that  are  capable  of 
generating  the  invariant.  By  sharply  distinguishing  the  general  from  the  particular,  such 
a  comparative  analysis  of  specialized  models  helps  to  discover  which  model  is  best  able 
to  explain  particular  parametric  behavior  about  brain  and  behavior,  or  to  solve  a  partic¬ 
ular  technological  problem.  Thus,  after  such  a  general  design  principle  or  computational 
invariant  is  discovered,  representative  models  may  be  reorganized  in  a  way  that  pays  less 
attention  to  accidents  of  their  historical  discovery  and  development  than  to  their  functional 
and  computational  properties.  The  identification  of  such  design  principles  and  computa¬ 
tional  invariants  is  thus  an  important  goal  of  computational  neuroscience. 

2.  Content-Addressable  Memory  and  Liapunov  Methods 

The  present  chapter  describes  a  general  design  principle  for  constructing  neural  net¬ 
works  that  are  capable  of  content-addressable  memory,  or  CAM.  From  a  mathematical 
perspective,  the  question  of  content-addressable  memory  in  a  m  ural  network  can  be  for- 


mulated  as  follows:  Under  what  conditions  does  a  neural  network  always  approach  an 
equilibrium  point  in  response  to  an  arbitrary,  but  sustained,  input  pattern?  The  equilib¬ 
rium  point  represents  the  stored  pattern  in  response  to  the  input  pattern.  In  a  satisfactory 
analysis  of  this  problem,  the  behavior  of  the  network  in  response  to  arbitrary  initial  data, 
an  arbitrary  sustained  input  pattern,  and  an  arbitrary  choice  of  network  parameters  is 
prox'ided.  Also  an  account  of  how  many  equilibrium  points  exist  and  of  how  they  are 
approached  through  time  is  desirable.  Such  a  mathematical  analysis  is  called  a  global 
analysis,  to  distinguish  it  from  a  local  stability  analysis  around  individual  equilibrium 
points. 

Amari  and  Arbib  (1982)  and  Levine  (1983)  have  described  a  number  of  contributions 
to  the  local  analysis  of  neural  networks.  Our  concern  herein  is  with  global  methods.  A 
global  analysis  of  equilibrium  behavior  is  of  importance  for  an  understanding  both  of  CAM 
and  of  the  types  of  nonequilibrium  behavior — such  as  travelling  waves,  bursts,  standing 
waves,  and  chaos  which  can  be  obtained  by  perturbing  off  systems  which  always  approach 
equilibrium  (Carpenter,  1977a,  1977b,  1979,  1981;  Cohen  and  Grossberg,  1983;  Ellias  and 
Grossberg,  1975:  Errnentrout  and  Cowan,  1979,  1980;  Hastings,  1976,  1982;  Hodgson,  1983; 
Kaczmarek  and  Babloyantz,  1977).  A  global  mathematical  analysis  of  nonlinear  associative 
learning  networks  was  begun  in  Grossberg  (1967,  1968b).  A  global  mathematical  analysis 
of  nonlinear  shunting  cooperative-competitive  feedback  networks  was  begun  in  Grossberg 
(1973).  Some  of  the  main  articles  in  these  series  are  brought  together  in  Grossberg  (1982). 

One  technique  for  the  global  approach  to  equilibrium  which  has  attracted  widespread 
interest  is  the  use  of  global  Liapunov,  or  energy,  methods.  Global  Liapunov  methods  were 
introduced  for  the  analysis  of  increasingly  large  classes  of  neural  networks  in  the  1970's 
(Grossberg.  1977,  1978a,  1978b,  1980,  1981). 

Prior  to  the  use  of  Liapunov  methods  for  the  analysis  of  neural  networks,  such  methods 
were  used  to  analyse  other  biological  networks,  notably  networks  arising  in  mathematical 
ecology.  For  example.  MacArthur  j  1970'  described  a  quadratic  Liapunov  function  for 
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proving  local  asymptotic  stability  of  isolated  equilibrium  points  of  Yolterra-Lotka  systems 
with  symmetric  coefficients.  Goh  and  Agnew  (1977)  described  a  global  Liapunov  function 
for  Volterra-Lotka  and  Gilpin- Ayala  systems  in  cases  where  only  one  equilibrium  point  ex¬ 
ists.  Liapunov  functions  were  also  described  for  Volterra-Lotka  systems  whose  off-diagonal 
terms  are  relatively  small  (Kilmer,  1972;  Takeuchi.  Adachi.  and  Tokumaru.  1978).  Such 
constraints  were,  however,  too  limiting  for  the  design  of  CAM  systems  aimed  at  trans¬ 
forming  and  storing  a  large  variety  of  patterns.  Herein  I  summarize  a  general  model  of 
a  nonlinear  cooperative-competitive  neural  network  for  which  a  global  Liapunov  function 
was  explicitly  constructed  to  ensure  CAM.  I  then  show  that  a  number  of  popular  models 
are  special  cases  of  the  general  model,  and  thus  are  capable  of  CAM. 

Cohen  and  Grossberg  (1983)  described  a  general  principle  for  designing  CAM  networks 
by  proving  that  models  that  can  be  written  in  the  form 


d-xt  =  at(xt)[fe,(jt)  -  ctjdj( j-j)] 

3  1 

admit  the  global  Liapunov  function 


V  =  ~llf  M£.K«U«Msi  +  \  YL  cjkdJ(xJ)dk(xk) 


if  the  coefficient  matrix  C  =||  ct)  ||  and  the  functions  a,,  bt.  and  dj  obey  mild  technical 
conditions,  including 


Symmetry: 


Positivity: 


Monotonicity: 


a,(r,)  >  0 


<!'}ls,)  >  0. 


1 


Integrating  V  along  trajectories  implies  that 


-V  =  -£>,<[6,  -  £  ctJdrf 


dt 


(6) 


t=i 


j=i 


If  (4)  and  (5)  hold,  then  j(  V  <  0  along  trajectories.  Once  this  basic  property  of  a  Liapunov 
function  is  in  place,  it  is  a  technical  matter  to  rigorously  prove  that  every  trajectory 
approaches  one  of  a  possibly  large  number  of  equilibrium  points. 

For  expository  vividness,  the  functions  in  the  Cohen-Grossberg  model  (1)  are  called 
the  amplification  function  a,,  the  self-signal  function  and  the  other-signal  functions  d}. 
Specialized  models  are  characterized  by  particular  choices  of  these  functions. 


3.  Additive  Model 

The  systematic  physical  and  mathematical  development  of  the  additive  model  began 
in  the  late  1960’s;  e.g..  in  Grossberg  (1967,  1968a,  1968b,  1969a.  1969b,  1970a,  1970b)  and 
Grossberg  and  Pepe  (1971).  Cohen  and  Grossberg  (1983,  p.819)  noted  that  “the  simpler 
additive  neural  networks  . . .  are  also  included  in  our  analysis”.  The  additive  equation  can 
be  written  using  the  coefficients  of  the  standard  electrical  circuit  interpretation  (Plonsey 


Thus  in  the  additive  case,  the  amplification  function  (8)  is  a  positive  constant,  hence 
satisfies  (4),  and  the  self-signal  term  (9)  is  linear.  Substitution  of  (8)-(ll)  into  (2)  leads 
directly  to  the  equation 


V  =  E  ft-  r  -  E  /,/,(*,)  -  \  E  Tjkfj{xj)fk(xk).  (12) 

t=l  1  t=l  “  j,k=  1 

This  Liapunov  function  for  the  additive  model  was  later  published  by  Hopfield  (1984). 
In  Hopfield’s  treatment,  is  written  as  an  inverse  (Iq).  Cohen  and  Grossberg  (1983) 
showed,  however,  that  although  f,(xt)  must  be  nondecreasing,  as  in  (5),  it  need  not.  have 
an  inverse  in  order  for  (12)  to  be  valid. 


4.  Brnin-State-in-a-Box  Model:  S  5Z  Exchange 

The  BSB  model  was  introduced  in  Anderson,  Silverstein,  Ritz,  and  Jones  (1977).  It  is 
often  described  in  discrete  time  by  the  equation 


x,{t  +  1)  =  5(xt(<)  +  a  E 

3  =  1 

using  symmetric  coefficients 

Atj  =  Ajj 


(13) 


(14) 


and  a  special  type  of  nonlinear  signal  function  5(u  )  that  characterizes  the  model.  The 
signal  function  is  a  symmetric  ramp  function: 

(  F  if  u:  >  F 

S(w)  —  <  w  if  —F  <  w  <  F  .  (15) 

(  -F  if  w  <  -F 

Thus  each  STM  trace  x,  obeys  a  linear  equation  until  its  argument  reaches  the  hard 
saturation  limit  F. 


The  BSB  model  has  been  used  to  discuss  categorical  perception  in  terms  of  its  formal 
contrast  enhancement  property  that  each  x,  tends  to  approach  a  limiting  value  ±F.  and 
thus  that  the  vector  ( X| .  x, . j-„)  tends  to  approach  a  corner  of  the  box  (rf.  tF . ±  F) 
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as  tune  goes  on.  An  alternative  explanation  of  contrast  enhancement  by  a  nonlinear  feed¬ 
back  network  was  provided  in  Grossberg  (1973)  using  a  sigmoid  signal  function,  rather 
than  a  function  linear  near  zero,  coupled  to  the  soft  saturation  dynamics  of  a  shunting 
network,  rather  than  the  hard  saturation  of  a  symmetric  ramp.  This  is  still  a  topic  un¬ 
dergoing  theoretical  discussion  (Anderson,  Silverstein.  Ritz,  and  Jones.  1977:  Grossberg, 
1978c,  1987d). 

The  BSB  model  can  be  rewritten  as  an  additive  model  with  no  input  and  a  special 
signal  function  that  satisfies  (5).  Hence  it  is  a  special  case  of  model  (1).  To  see  this,  rewrite 
(13)  in  the  form 


using  the  coefficient 


x t{t  +  l)  —  •S' ( x}{t)) 

j=i 


Btj  —  6tj  +  aAtJ 


where  StJ  =  1  if  i  =  j  and  0  if  i ^  j.  By  (14),  it  follows  that 


BtJ  —  BJt. 


Although  (16)  is  written  in  discrete  time  for  computational  convenience,  it  needs  to  be 
expressed  in  continuous  time  in  order  to  represent  a  physical  model,  as  in 


jtxt  =  -x,  +  S^B.jXj). 

j=i 


Define  the  new  variables  yt  by 


Then 


Vi  ~  BijXy 


dt  Vr  ~  ~Vi  +  E  (21) 

1 

Comparison  of  (21)  with  (7)  shows  that  the  BSB  model  is  an  additive  model  such  that 
each  input  I,  =  0.  This  simple  change  of  coordinates  from  (19)  to  (21)  is  so  important  in 
neural  modelling  that  I  give  it  a  name:  Signal-Sutri  /'jZJ  Exchange. 


WWW* 


r.r.r-r,  »•,  *■  ,*  .  *  .•»'»*  , 


The  observation  that,  via  S  ^  Exchange,  a  nonlinear  signal  of  a  sum.  as  in  (19),  can 
be  rewritten  as  a  sum  of  nonlinear  signals,  as  in  (21).  shows  that  a  number  of  models 
which  have  been  treated  as  distinct  are  mathematically  identical.  In  contrast,  this  type  of 
transformation  cannot  be  carried  out  on  shunting  models  such  as  (31)  below. 

The  Liapunov  function  for  (21)  is  found  by  directly  substituting  into  model  (1)  ex¬ 
pressed  in  terms  of  the  variables  y,: 

d  n 

jtyx  =  a,-(y,-)[M»i)  -  Z cijdi (vj)l-  (22) 

>=i 

Since  a,(yt)  =  1,  &,(y,)  —  —  y,,  ctj  =  -Blv  and  dj(yj)  —  S(y}),  substitution  into  (2)  yields 

V  =  tr  \  Z  B}kS(yj)S(yk).  (23) 

«=r  L  Jt=i 

Lsing  the  definitions  in  (15),  (17),  and  (20),  (23)  can  be  rewritten  in  terms  of  the  original 
variables  xt  as  follows: 

y  =  ~2  Z  AjkXjXk.  (24) 

>,*=J 

Golden  (1986)  has  derived  (24)  from  a  direct  analysis  of  the  BSB  model. 

5.  The  McCulloch-Pitts  Model 

This  classical  model  takes  the  form 


n 

xt(t  +  1)  =  sgn(£  A,jXj(t)  -  Bx).  (25) 

]=\ 

Letting 

M(w)  -  sgn(; v  -  Bt),  (26) 

(25)  can  be  rewritten  as 

*.('  +  1)  =-V/(Z  Atjxj(t)).  (27) 

t=i 

As  in  the  analysis  of  (19).  (27)  can  be  rewritten  in  continuous  time  in  terms  of  the  variables 
y,  via  .V  ^  Exchange: 

d  " 

(jt y,  ~-y,  +  AtJM(yj) 

i  i 


(2S) 
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and  is  tlms  also  a  symmetric  additive  model  with  zero  inputs.  In  addition,  its  signal 
function  XI ( xjj )  has  a  zero  derivative  (Xl'(yj)  =  0)  except  at  y 3  —  0.  Substitution  of  this 
additional  property  into  (23)  shows  that  the  Liapunov  function  for  the  continuous  time 


McCulloch-Pitts  model  is 


V  =  -o  E  A]kM(yj)M(yk), 


which  is  the  continuous  time  version  of  the  discrete  time  Liapunov  function  described  by 
Hopfield  (1982). 

6.  The  Boltzmann  Machine 

The  state  equation  of  the  Boltzmann  machine  (Ackley,  Hinton,  and  Sejnowski,  1985) 
is  also  an  additive  equation  with  symmetric  coefficients.  Its  signal  function  is  the  sigmoid 
logistic  function 

/H  =  x-+U  (30) 

which  satisfies  (5)  and  is  thus  a  special  case  of  model  (1).  Thus  the  Boltzmann  machine 
is  a  specialized  additive  model  regulated  by  simulated  annealing  (Geman.  1983,  1984; 
Kirkpatrick,  Gelatt,  and  Vecchi,  1982). 

7.  Shunting  Cooperative-Competitive  Feedback  Network 

All  additive  models  lead  to  constant  amplification  functions  a, (x,)  and  linear  self¬ 
feedback  functions  6,(i,).  The  need  for  the  more  general  model  (1)  becomes  apparent  when 
the  shunting  model  is  analysed.  Consider,  for  example,  a  class  of  shunting  models  in  which 
each  node  can  receive  excitatory  and  inhibitory  inputs  /,  and  J,.  respectively,  and  each 
node  can  excite  itself  and  can  inhibit  other  nodes  via  nonlinear  feedback.  Such  networks 
model  on-center  off-surround  interactions  among  cells  which  obey  membrane  equations 
(Grossberg.  1973;  Hodgkin.  19G4;  Kandel  and  Schwartz.  1981;  Katz,  1966;  Plonsey  and 
Fleming.  I960).  In  particular,  let 

d  " 

a:-!-,  -  -A,x,  +  (D,  -  x,)!/,  +  -  (x,  +  C,)\J,  +  Dnyt(i})).  (31) 


3 


In  (31).  each  x,  can  fluctuate  within  the  finite  interval  [  — C,.  Bt j  in  response  to  the  constant 
inputs  I,  and  J, ,  the  state-dependent  positive  feedback  signal  /, (x,).  and  the  negative 
feedback  signals  DX]g](x]).  It  is  assumed  that 


Dtj  =  DJt>  0 


(32) 


and  that 


gj(Xj)  >  0. 


(33) 


In  order  to  write  (31)  in  Cohen-Grossberg  form,  it  is  convenient  to  introduce  the  variables 


y,  =  x,  +  C,. 


(34) 


In  applications.  C,  is  typically  nonnegative.  Since  x,  can  vary  within  the  interval  [— C,.  Z?,j. 
t/j  can  vary  within  the  interval  [0,  Bt  +  C,]  of  nonnegative  numbers.  In  terms  of  these 
variables,  (31)  can  be  written  in  the  form 


where 


d  n 

jtvi  ~  «t(ya)lMyt)  —  Cijd](yj)} 


a,  (y,)  =  y,  (nonconstant!). 


(22) 


(35) 


1 


K(yt)  =  —  [<4,C,  -  (.4,  4-  j,)y,  +  (B,  +  Ct  -  y,)(I,  +  /,(y,  -  C,))]  (nonlinear!),  (36) 


CXJ  =  DXJ . 


(37) 


and 


dj(yj)  =  gj{yj  -  Cj)  (noninvertible!). 


(38) 


Unlike  the  additive  model,  the  amplification  function  a,(y,)  in  (35)  is  not  a  constant.  In 
addition,  the  self-signal  function  b,  (y,)  in  (36)  is  not  necessarily  linear,  notably  because 
the  feedback  signal  /,(x,  -  (',)  is  often  nonlinear  in  applications  of  the  shunting  model:  in 
particular  it  is  often  a  sigmoid  or  multiple  sigmoid  signal  function  (Ellins  and  Grossberg. 


1973:  Grossberg.  1973.  1977.  1978b;  Grossberg  and  Levine.  1975;  Sperling.  1981).  Sigmoid 
signal  functions,  and  approximations  thereto,  also  appear  in  applications  of  the  additive 
model  and  its  variants  (Ackley,  Hinton,  and  Sejnowski,  1985;  Amari  and  Arbib,  1982: 
Freeman.  1975.  1979:  Grossberg,  1969a,  1982;  Grossberg  and  Kuperstein,  1986;  Hinton 
and  Anderson.  1981;  Hopfield.  1984;  Rumelhart  and  McClelland,  1986).  Such  applications 
do  not  require  the  full  generality  of  the  Liapunov  function  (l)  because  the  nonlinear  signal 
function  can  then  be  absorbed  into  the  terms  dj(xj). 

Property  (4)  follows  from  the  fact  that  a,( yt)  =  y,  >  0.  Property  (5)  follows  from 
the  assumption  that  the  negative  feedback  signal  function  g}  in  (38)  is  monotone  non- 
decreasing.  Cohen  and  Grossberg  (1983)  proved  that  g}  need  not  be  invertible.  A  signal 
threshold  may  exist  below  which  g}  ~  0  and  above  which  g}  may  grow  in  a  nonlinear  way. 
The  inclusion  of  nonlinear  signals  with  thresholds  better  enables  the  model  to  deal  with 
fluctuations  due  to  subthreshold  noise.  On  the  other  hand,  thresholds  are  not  the  only 
mechanisms  which  can  suppress  noise  in  a  cooperative-competitive  feedback  network. 

8.  Masking  Field  Model 

In  many  applications' of  the  shunt  ing  and  additive  models,  the  coefficients  cl3  in  (1) 
may  be  asymmetric,  thereby  rendering  the  Liapunov  function  (2)  inapplicable.  Asym¬ 
metric  coefficients  typically  occur  in  problems  relating  to  the  learning  and  recognition 
of  temporal  order  in  behavior.  Consequently,  a  number  of  mathematical  methods  have 
been  developed  from  the  earliest  days  of  neural  network  theory  to  analyse  models  with 
asymmetric  interaction  coefficients. 

On  the  other  hand,  certain  network  models  may  have  asymmetric  interaction  coeffi¬ 
cients,  yet  be  reduceable  to  the  form  (1)  with  symmetric  interaction  coefficients  through  a 
suitable  change  of  variables.  The  masking  field  model  is  a  shunting  network  of  this  type. 
The  masking  field  model  was  introduced  in  Grossberg  (l97Sd:  reprinted  in  Grossberg, 
1982)  to  explain  data  about  speech  learning,  word  recognition,  and  the  learning  of  adap- 


tive  sensory- mot  or  plans.  It  has  been  further  developed  through  computer  simulations  in 
Cohen  and  Grossberg  ^  1986.  1987).  A  masking  field  is  a  multiple-scale,  self-similar,  auto¬ 
matically  gain  controlled,  cooperative-competitive  nonlinear  feedback  network  which  can 
generate  a  compressed  but  distributed  STM  representation  of  an  input  pattern  as  a  whole, 
of  its  most  salient  parts,  and  of  predictive  codes  which  represent  larger  input  patterns  of 
w'hich  it  forms  a  part.  The  masking  field  model  is  thus  a  specialized  type  of  vector  quanti¬ 
zation  scheme  (Gray.  1984).  Its  multiple-scale  self-similar  properties  imply  its  asymmetric 
interaction  coefficients. 

The  STM  equation  of  a  typical  masking  field  is  defined  by 

p,J)  =  -  MJ)  +  {B-  EjrW  +  D\J\ /(x‘J|)] 

)f:J  (39) 

,  (J)  ,  ^m,Kg(^])\K\{i+\KnJ\) 

l  <  j  Em,K  \K  I  (1+  \K  C  J  |)  • 

In  (39),  is  the  STM  trace  of  the  tth  masking  field  node  that  receives  excitatory  input 
eJ  EjP\i  *  from  the  unordered  set  J  of  input  items.  Notation  |  J  |  counts  the  number 
of  items  in  set  J  and  thereby  keeps  track  of  the  number  of  spatial  scales  that  go  into  each 
version  of  the  model. 

The  inhibitory  interaction  coefficient 


K  I  (i+  I  A'n  J  I) 

E«,k  |  K  I  (i+  I  K  n  J  I) 

in  (39)  is  an  asymmetric  function  of  J  and  K.  Despite  this  fact,  (39)  can  be  written  in 
Cohen-Grossberg  form  as 


£  cKJ<l{h\ym  ’)] 
m.K 


(41) 


with  symmetric  coefficients 


in  terms  of  the  variables 


where 


£  I  lfMi+1  A-nj|).  (44) 

m.K 

This  is  seen  as  follows.  Since  F j<  is  the  denominator  of  (40).  it  can  be  used  to  divide 
term  +  C  in  (39).  Then  the  asymmetric  term  |  A’  |  in  the  numerator  of  (40)  can  be 
absorbed  into  the  definition  of  g  in  (39).  Then  by  redefining  and  rearranging  terms  as  in 
(35)-(38).  equation  (41)  holds  with 

(«) 

+  (46) 

4-  D  I  J  I  iv,/(lvl!,|J|  -  C))| 


where 


<J)  =  Frn  E  EiPV\ 

J£J 


<fiK)(y£))=\K\g(F,Kyi*)-c). 


Thus  the  masking  field  model  is  a  specialized  Cohen-Grossberg  model. 


9.  Bidirectional  Associative  Memories:  Symmetrizing  an  Asymmetric  In¬ 
teraction  Matrix 

Other  procedures  have  also  been  devised  for  dealing  with  systems  having  asymmetric 
coefficients;  for  example,  given  an  arbitrary  n  x  m  coefficient  matrix  Z  =|  zt]  |  from  a 
network  level  F\  to  a  network  level  Fi  with  STM  traces  x,  and  y}.  respectively.  Kosko 
and  Guest.  (1987)  and  Kosko  (1987)  have  shown  that  (1)  and  (2)  ran  be  used  to  construct 
feedback  pathways  from  F2  to  Fj  so  that  the  two-level  feedback  network  F\  <—  F 2  has 
convergent  trajectories. 


For  example,  if  the  bottom-up  interaction  F\  -*  F 2  obeys  an  additive  equation 


jt'jj  -  ~Ajyi  +  YLfk(xk)=k]  + 1],  (4Q) 

k 

then  the  top-down  interaction  F2  —  Fx  is  defined  to  obey  an  additive  equation 

=  -Bi xt  +  Y  9i(yi)zti  +  Jt-  (50) 

where  Ij  and  J,  are  input  terms.  This  definition  creates  a  symmetric  interaction  matrix 
by  dosing  the  top-down  feedback  loop,  since  if  ft{xt)  influences  with  coefficient  zt]  in 
(49),  then  </_,(*/_,)  influences  x ,  with  the  same  coefficient  ztJ.  Thus  by  defining  an  augmented 
vector  ( xi ,  z2>  •  •  •  >  xn,  Vi ,  1/2 1  •  •  • ,  Um)  of  STM  activities,  system  (49)-(50)  as  a  whole  defines 
an  additive  model  (7)  with  an  (n  +  m)  x  (n  +  m)  symmetric  coefficient  matrix. 

The  same  procedure  can  be  used  to  symmetrize  many  other  neural  network  models. 
Kosko  and  Guest  (1987)  have  described  optical  implementations  for  this  procedure,  and 
Kosko  (1987)  has  used  the  symmetrized  additive  model  to  discuss  minimization  of  fuzzy 
entropy. 

10.  Volterra-Lotkn.  Gilpin-Ayala,  and  Eigen-Sclmster  Models 

The  Cohen-Grossberg  model  was  designed  to  also  include  models  which  arose  in  other 
areas  of  biology  than  neural  network  theory.  For  example,  it  includes  the  classical 

Volterra-Lotkn  Model 

~x,  =  .4,xt(l  -  Y  (51) 

J=1 

of  population  biology  (Lotka.  1956).  the 

Gilpin-Ayala  Model 


also  from  population  biology  (Gilpin  and  Ayala.  1973).  and  the 


Eigen-Schuster  Model 


~x,  =  x,(Atx?  1  ~q)T  A}xp})  (53) 

j-i 

from  the  theory  of  macromolecular  evolution  (Eigen  and  Schuster,  1978).  In  all  of  these 
models,  either  the  amplification  function  a,(x,)  is  non-constant,  or  the  self-signal  function 
6,(x,)  is  nonlinear,  or  both. 

11.  Concluding  Remarks:  Comparative  Analysis  and  Model  Selection 

The  specialized  models  summaried  in  Sections  2-9  illustrate  that  model  (l)  and  Lia¬ 
punov  function  (2)  embody  a  general  principle  for  designing  CAM  devices  from  coopera¬ 
tive-competitive  feedback  models.  These  models  are  said  to  be  absolutely  stable  because 
the  CAM  property  is  not  destroyed  by  changing  the  parameters,  inputs,  or  initial  values 
of  the  model.  The  persistence  of  the  CAM  property  under  arbitrary  parameter  changes 
enables  learning  to  change  system  parameters  in  response  to  unpredictable  input  envi¬ 
ronments  without  destroying  CAM.  Nonetheless,  the  transformation  from  input  patterns 
to  stored  patterns  executed  by  a  network  with  adaptively  altered  parameters  can  differ 
significantly  from  its  original  transformation. 

The  Cohen-Grossberg  analysis  emphasizes  the  critical  role  of  mathematical  methods 
in  classifying  and  understanding  very  large  systems  of  nonlinear  neural  networks  (VLSN). 
Without  such  an  integrative  approach,  it  is  sometimes  difficult  to  tell  even  whether  or 
not  a  model  is  really  new  computationally,  or  whether  it  is  a  variant  of  a  known  model 
in  different  coordinates  or  notation.  For  example,  many  scientists  have  not  realized  that 
models  (19)  and  (21)  are  mathematically  equivalent.  Table  1  describes  the  relationships 
between  models  summarized  herein  by  surh  an  analysis.  Thus  the  Brain-State-in-a-Box 
and  Boltzmann  machine  models  enjoy  a  CAM  property  for  the  same  reason  that  any 
additive  or  Cohen-Grossberg  model  does.  On  the  other  hand,  the  BSB  model  may  have 
special  properties  that  may  make  it  ideal  for  certain  ta-ks.  or  it  may  be  too  specialized  to 


accomplish  certain  tasks  which  are  better  dealt  with  using  a  more  general  additive  model, 
or  even  a  shunting  model. 

Table  1 

The  set  of  specialized  properties  which  differentiate  one  model  from  another  ultimately 
provides  the  computational  rationale  for  continued  interest  in  that  model,  whether  because 
its  set  of  properties  better  explain  an  important  behavioral  or  neural  data  base,  or  because 
these  properties  enable  the  model  to  more  efficiently  solve  an  important  technological 
problem.  Indeed,  the  Cohen-Grossberg  (1983)  model  was  developed  to  integrate  many  of 
the  specialized  additive  and  shunting  models  which  had  arisen  in  applications  since  the  late 
1960’s.  Once  such  an  integrative  design  is  recognized,  a  comparative  analysis  of  specialized 
models  is  mandated  within  the  computational  framework  defined  by  the  design  principle 
and  its  unifying  mathematical  method.  Such  a  comparative  analysis  needs  to  clearly 
distinguish  between  the  model-independent  properties  that  are  shared  by  all  models  which 
exemplify  the  design  and  the  unique  combinations  of  model-dependent  properties  which 
both  the  evolutionary  process  and  the  human  engineer  seek  to  utilize  to  their  maximal 
advantage. 
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